The present invention relates to an image sensing apparatus and, more particularly, to an image sensing apparatus, using a non-interlace scanning type image sensing device, capable of encoding image data obtained by the image sensing device in accordance with movement of an object sensed, and outputting a smooth moving image when reproducing image data recorded on a recording medium.
Recently, a non-interlace scanning type image sensing device capable of sequentially reading signals of all the pixels has been developed with the progress of semiconductor manufacturing technique.
The non-interlace scanning type image sensing device has an advantage in that a higher resolution image can be obtained with less blurring than an image sensed by using a conventional interlace scanning type image sensing device even when sensing a moving object.
In the interlace scanning type image sensing device, a frame image is composed of two field images which are sensed at different times, usually at a field period interval. Accordingly, there is a problem in which, when sensing a fast moving object, there are notches on edges of the object and perhaps of the background in a frame image because of the time gap between the two field images composing a frame image.
If a frame image is made of image data of a single field image to overcome the aforesaid problem, there would not be notches on edges, however, since the amount of image information in the vertical direction is halved compared to a frame image composing of two field images, the vertical resolution of the obtained frame image is also halved.
In contrast, with a non-interlace scanning type image sensing device, it is possible to sense a frame image in the same time period as that for sensing a field image by an interlace scanning type image sensing device, thus, the above problem does not arise. By taking this advantage of the non-interlace scanning type image sensing device, it is applied to a still image camera and an input device for use with a computer, for example.
Further, in a still image output device, such as a video printer, which has rapidly spread in the market in these days, a user can arbitrary pick up a desired scene out of images which are sensed as a moving image. Accordingly, there is a demand to use the non-interlace scanning type image sensing device as an image sensing unit of a video camera capable of sensing both a moving image and a still image.
When a non-interlace scanning type image sensing device is used as an image sensing unit of a video camera capable of sensing both a moving image and a still image, as described above, a couple of methods for generating moving image signals can be considered.
A case where a non-interlace scanning type image sensing device is used in a digital video camera of NTSC standard, as shown in FIG. 7, will be explained as an example. A non-interlace scanning type image sensing device 1 has a structure to output signals by two channels, and each channel always outputs either image signals of even lines or image signals of odd lines of the non-interlace type image sensing device 1.
Further, in FIG. 7, image signals of the even lines and odd lines are alternatively outputted from each channel of the non-interlace scanning type image sensing device 1 in each field period in accordance with timing signals generated by a timing signal generator (TG) 15. For example, referring to one of the two output channels, when image signals of even lines are outputted from one of the channel in a given field period, image signals of odd lines are outputted in the next field period, then image signals of even lines are outputted in the following field period. Image signals read out from the non-interlace scanning type image sensing device 1 are respectively inputted to correlated double sampling (CDS) circuits 201 and 202. The signals outputted from the CDS circuits 201 and 202 are inputted to automatic gain controllers (AGCs) 301 and 302, thereafter enter analog-digital (A/D) converters 401 and 402, respectively.
Then, after the analog signals are converted into digital signals by the A/D converters 401 and 402, enter a camera signal processing circuit 5. The camera signal processing circuit 5 performs signal processes, such as color separation, edge enhancement, and color correction, after the image data of the even lines and odd lines are applied with dot sequential processing.
After the aforesaid processes are completed, the camera signal processing circuit 5 divides a frame image, and image data of one field (e.g., image data of even lines) is outputted from the first channel ch1, and image data of the other field (e.g., image data of odd lines) is outputted from the second channel ch2. Similarly, for the next frame image, image data of alternate fields are outputted from the first and second channels ch1 and ch2. For example, image data of odd lines is outputted from the first channel ch1, and image data of even lines is outputted from the second channel ch2.
The non-interlace scanning type image sensing device 1 can generate a frame image in one field period, however, a recording device (e.g., a digital VTR) can record only a field image in one field period.
Accordingly, as shown in FIG. 8A, by using either the image signals outputted from the first channel ch1 or the image signals outputted from the second channel ch2, an image of a single field is outputted in each field period (a mode for performing the aforesaid operation is called xe2x80x9cfield image sensing modexe2x80x9d, hereinafter).
Referring to FIG. 8A, the camera signal processing circuit 5 sequentially writes field image data of a first frame image #1 and of a second frame image #2 outputted from the first channel ch1 to a first frame memory 601 in the first and second field periods in accordance with a control signal C1.
Meanwhile, if image data is written in a second frame memory 602, the image data of one previous frame period is sent to an encoding processing circuit 7 in accordance with a control signal C2. In the encoding processing circuit 7, the image data is applied with processes, such as discrete cosine transform (DCT) and shuffling, thereafter, stored in a magnetic tape 9 by a recording head 8 as digital image signals in a recording method complying with a format.
After image data of one frame is written in the first frame memory 601, the camera signal processing circuit 5 sequentially writes field image data of the third frame image #3 and of the fourth frame image #4 outputted from the first channel ch1 to the second frame memory 602 in the third and fourth field periods in accordance with the control signal C2.
Meanwhile, the image data of one previous frame period is sent to the encoding processing circuit 7 in accordance with a control signal C2. In the encoding processing circuit 7, the image data is applied with the same processes as described above, thereafter, stored in the magnetic tape 9 by the recording head 8 as digital image signals in the recording method complying with the format.
The image signals which are recorded as above is processed as shown in FIG. 8B when they are reproduced. First, the image signals read from the magnetic tape 9 by the read head 10 are sent to a decoding processing circuit 11 in the first and second field periods, and applied with signal processes, such as inverse discrete cosine transform (I-DCT) and de-shuffling. Thereafter, the image signals are written to a third frame memory 121 in accordance with a control signal C3.
Meanwhile, if image data is written in a fourth frame memory 122, the decoding processing circuit 11 reads image data from the fourth memory 122 in accordance with a control signal C4, and reproduces an image by an even line field and an odd line field separately.
In the third and fourth field periods, the image signals read from the magnetic tape 9 by the read head 10 are transmitted to the decoding processing circuit 11 where the image signals are applied with the I-DCT, deshuffling, and so on, then written to the fourth frame memory 122.
Meanwhile, the decoding processing circuit 11 reads image data from the third memory 121 in accordance with the control signal C3, and reproduces an image of the even line field and the odd line field separately.
In the decoding processing circuit 11 used in the aforesaid conventional example, image signals outputted from the first channel ch1 and the second channel ch2 are of the even line field and the odd line field of the non-interlace scanning type image sensing device 1, shown in FIG. 10. Therefore, the field image sensing mode can be used for moving image sensing operation.
However, according to the aforesaid example, the vertical resolution is about the same as that of a conventional image sensing device which outputs image data after adding charges stored in two adjacent pixels in the vertical direction. Accordingly, when outputting an image of a moving object obtained by using the non-interlace scanning type image sensing device 1 in the field image sensing mode as a still image from a video printer, or the like, only a poor still image can be obtained because of blurring.
Thus, as a method of effectively using an advantage of the non-interlace scanning type image sensing device, i.e., to output a frame image in a field period, the one shown in FIG. 9A has been suggested.
In this case, the camera signal processing circuit writes field image data of the even line field of the frame image #1 outputted from the first channel ch1 and field image data of the odd line field of the frame image #1 outputted from the second channel ch2 to the first frame memory 601 in the first and second field periods, respectively, in accordance with the control signal C1.
Then a frame image #2 sensed in the second field period is not stored, and the first frame image sensed in the first field period is used as a moving image of a frame period (this image sensing operation is called xe2x80x9cframe image sensing modexe2x80x9d, hereinafter).
Meanwhile, if image data has been written in the second frame memory 602, the image data of one previous frame is sent to the encoding processing circuit 7 in accordance with the control signal C2 and processed with DCT, shuffling, and so on. Thereafter, the processed image data is recorded as digital image signals on the magnetic tape 9 by the recording head 8 in a recording method complying with a format.
When image data of a single frame is written in the first frame memory 601, the camera signal processing circuit sequentially writes even line field image data of the third frame image #3 outputted from the first channel ch1 and odd line field image data of the third frame image #3 outputted from the second channel ch2 to the second frame memory 602 in the third and fourth field periods in accordance with the control signal C2.
Meanwhile, the image data of one previous frame written in the first frame memory 601 is send to the encoding processing circuit 7 where it is applied with the same processes as described above, then recorded on the magnetic tape 9 by the recording head 8 in a recording method complying with a format as digital image signals.
The image signals recorded as described above are applied with processes as shown in FIG. 9B when they are reproduced. First, the image signals read from the magnetic tape 9 by the read head 10 are transmitted to the decoding processing circuit 11 where they are processed with I-DCT, deshuffling, and so on. Then, the image signals are written to the third frame memory 121 in accordance with the control signal C3 in the first and second field periods.
Meanwhile, if image data is written in the fourth frame memory 122, the decoding processing circuit 11 reads image data from the fourth memory 122 in accordance with the control signal C4, and reproduces an image by an even line field and an odd line field separately.
In the third and fourth field periods, the image signals read from the magnetic tape 9 by the read head 10 are transmitted to the decoding processing circuit 11 where they are applied with the I-DCT, deshuffling, and so on. Thereafter, the images are written to the fourth frame memory 122 in accordance with the control signal C4.
Meanwhile, the decoding processing circuit 11 reads image data from the third memory 121 in accordance with the control signal C3, and reproduces an image using the even line field and the odd line field separately.
In this method, it is possible to store an image of high resolution without blurring when sensing a moving object. Therefore, the method can be used when sensing a still image. However, when the stored image data is reproduced as a moving image, image data of fields as shown in FIG. 9B is outputted, thus the displayed image is of frame images sensed in frame period. In this method, therefore, it is possible to obtain an image of high resolution, however, when the image data of a faster moving object is reproduced as a moving image, for example, the displayed image has gaps in time and only a poor moving image can be obtained.
The present invention is addressed to solve this problem.
Further, as the digital signal processing technique improves, many image sensing apparatuses adopting digital recording and reading technique in the recording and reproducing unit have been proposed. In these image sensing apparatuses, image signals are compressed and encoded as well as modulated to a format suitable for digital recording in a recording unit, then recorded in a data storage medium. Further, when reproducing image signals, read data is demodulated and decoded in a process in opposite to the recording process, then outputting reproduced image signals.
FIG. 11 is a block diagram illustrating a configuration of a conventional image sensing apparatus. In FIG. 11, reference numeral 501 denotes an image sensing unit whose focus, zoom ratio, and iris diaphragm, and so on, are controlled by an image sensing controller 503, and which generates known digital standard image data S1p, such as parallel data conforming with SMPTE (Society of Motion Picture and Television Engineers) 125M.
A block division unit 502 divides the digital image data S1p into blocks consisting of a plurality of pixels, further applies processes, such as shuffling and noise reduction, on the divided digital image data.
The image data S2p divided into a plurality of blocks by the block division unit 502 is provided to a motion detector (MD) 505.
The MD 505 generates information S3p on movement of image data on the basis of the input image data S2p and outputs it to a system controller 509. The MD 505 detects movement in an image by detecting differences between field image data of each image blocks.
Reference numerals 506 and 507 denote discrete cosine transform (DCT) units which compress information by using correlation between neighboring pixels of the image data. The first DCT unit 506 performs DCT on image data by an area, e.g., 8xc3x978 pixel block, of a frame image.
Further, the second DCT unit 507 performs DCT on image data by an area, e.g., by 8xc3x974 pixel block, of an odd line field image, and 8xc3x974 pixel block of an even line field image.
The system controller 509 outputs a switching signal S4p in accordance with information on movement, thereby controls a switch 508 to switch between the first DCT unit 506 and the second DCT unit 507.
Here, in a case where no movement is detected in the block image data S2p, in other words, movement determination information S3p shows xe2x80x9cnot movingxe2x80x9d, the switch 508 switches to the terminal 508a. In contrast, in a case where movement is detected in the block image data S2p, i.e., the movement determination information S3p shows xe2x80x9cmovingxe2x80x9d, the switch 508 switches to the terminal 508b. Thus, DCT processes can be switched for a frame image and for a field image in accordance with the movement determination information S3p.
In a case where there is a large movement in the block image data S2p, since correlation between fields is low, vertical components of the DCT coefficients reach high frequency range if an image is processed as a frame image, and encoding efficiency drops extremely.
Therefore, when the movement determination information S3p shows xe2x80x9cmovingxe2x80x9d, it is controlled so that odd line field image data and even line field image data are separately applied with orthogonal transformation. As described above, by properly switching the DCT between the one for a frame image and the one a field image in accordance with a state, xe2x80x9cmovingxe2x80x9d or xe2x80x9cnot movingxe2x80x9d shown by the movement determination information S3p, effective encoding is performed.
Data S5p processed with the DCT by the first DCT unit 506 or the second DCT unit 507 is quantized by a quantization unit 510. Processes at each step of quantization are adjusted in accordance with the precision of the image data, and an image of low frequency is quantized closely, whereas an image of high frequency is quantized roughly.
This is because distinguishable ability of human eyes is keen for an image of low frequency, in contrast, it is dull for an image of high frequency. Therefore, by quantizing image data of low frequency range closely and image data of high frequency range roughly, distortion of an image caused by the quantization is concentrated on the high frequency components, thereby reducing deterioration of a visual image quality.
An encoding unit 511 scans the block data arranged in two dimension in zig-zag scanning from the low space frequency portion to the high space frequency portion to obtain linear data, encodes zero coefficients by run-length coding and non-zero coefficients by two dimensional Huffman coding into variable length codes, then outputs encoded data S6p.
In the run-length coding, image data is applied with lossless compression in accordance with a zero-run count. In the Huffman coding, short codes are assigned to data whose occurrence probability is high, whereas long codes are assigned to data whose occurrence probability is low, thereby shortening the total code length.
A flag controller 512 is for generating a system information flag S7p used when writing information outputted from the system controller 509 to a recording medium.
In an image sensing apparatus adopting digital recording and reading technique, not only a moving image but also a still image can be recorded in high precision. Further, it is possible to record a still image of high precision (still image recording mode) while recording a moving image (moving image recording mode) depending upon an image sensing mode.
The conventional image sensing apparatus as described above determines movement based on correlation of block image data between fields, a proper DCT method performed by frame or by field is selected on the basis of the detected result.
However, upon sensing and recording a still image of high precision by using a progressive or non-interlace scanning type CCD, since information on the vertical resolution of the image contains higher frequency components than that of an image sensed by using an interlace scanning type CCD, it becomes very difficult to determine movement. For example, when a fine stripe pattern is in an image and differences between even line field data and odd line field data of the image are calculated to be used for detecting movement, since the differences between the field data would be large because of the high vertical resolution, there would be more chance for the image to be misjudged as a moving image. Accordingly, there are more cases in which a still image is misjudged as a moving image, thereby encoding efficiency drops.
The present invention is also addressed for solving the above problem.
The present invention has been made in consideration of aforesaid situation, and has as its object to obtain a moving image of natural movement and of high resolution even in a case where frame images are recorded by using a non-interlace or progressive scanning type image sensing device.
According to the present invention, the foregoing object is attained by providing an image sensing apparatus which has a reproduction function comprising:
image sensing means for generating image signals by sequentially reading all the pixels of the image sensing means in non-interlace scanning in one field period; camera signal processing means for generating first image signals and second image signals based on television signal standard; output signal selection means for selecting the first image signals or the second image signals outputted by the camera signal processing means; signal recording means for recording a type signal showing a type of the image signals selected by the output signal selection means in a sub-cord recording area of a recording medium;
an interpolation filter for interpolating image data between consecutive images sensed at different times; and interpolation filter control means for controlling whether or not to perform interpolation of image data using the interpolation filter, wherein, when the type signal recorded in the sub-cord recording area indicates a frame image sensing mode in which frame images are sensed in every other field period and the first image signals are recorded in a given field period and the second image signals are recorded in the next field period, and when the sensed image signals are to be outputted as a moving image, the interpolation filter control means controls the interpolation filter to perform interpolation of image data.
According to the aforesaid configuration, when a moving image is sensed in the frame image sensing mode which is essentially suitable for still image sensing, by interpolating field images between consecutive frame images sensed at different times, it is possible to make a moving image continue smoothly. Therefore, if all the recorded image data is of still frame images, it is possible to display it as a moving image of high resolution continuing naturally.
It is another object of the present invention to prevent drop of coding efficiency caused by misdetection of movement in a still image when coding the still image of high precision obtained by non-interlace scanning.
According to the present invention, the foregoing object is attained by providing an image sensing apparatus comprising: image sensing means for generating image signals by sequentially reading all the pixels of the image sensing means in non-interlace scanning in one field period; image sensing mode selection means for selecting an image sensing mode performed by the image sensing means out of a plurality of image sensing modes; and movement determination means for performing movement determination processes in accordance with the image sensing mode selected by the image sensing mode selection means.
With the aforesaid configuration, upon sensing an image by using an image sensing device which sequentially outputs signals generated at each pixel without adding the signals, movement is detected in accordance with the image sensing mode, i.e., a moving image sensing mode or a still image sensing mode. Thereby, it is possible to reduce a chance to misdetect the movement in a still image caused by different features between the moving and still image sensing modes.